Spg. til teksten?

• I modsætning til traditionel klyngeanalyse, er ideen i topic modeling at hvert dokument består af flere forskellige topics i forskellige proportioner, fremfor at hvert dokument typisk kun tilhører én klynge.

• Et dokument kan derfor tilhøre flere emner på én gang, med forskellig styrke eller sandsynlighed.

• En form for “specialiseret klyngeanalyse”.

• Hvert topic er “blot” en samling af ord, hvor hvert ord har en estimeret sandsynlighed, der indikerer hvor sandsynlige deres fremkomst er i en tekst med et givent topic.

• Topics kan også tænkes i “bag”-analogien: hvert tema er en “bag” af ord, hvor nogle er meget fremkomne i én “bag”, men mindre sandsynlige i en anden “bag”.

  • “Klima” er meget sandsynlig i vores “miljø”-topic, men fremkommer også i vores “skole”-topic, endog fremkomsten af ordet her er mindre sandsynligt en det første topic.

LDA er iterativ, idet det er en proces, hvor algoritmen laver en gradvis “forbedring” af resultaterne (fordelingerne)—gører tilpasningsdelen igen og igen—indtil den finder en stabil løsning (konvergerer), hvor hver gentagelse kun ændrer resultaterne minimalt. Ved dette punkt har algoritmen har fundet et stabilt sæt af topics.

~~Stiliseret iterativ algoritme~~ 
Input: Et sæt af dokumenter D, antal ønskede topics K
Output: Document-topic-fordeling og Topic-word-fordeling

Start:
    Initialisér tilfældigt et topic for hvert ord i alle dokumenter

    Gentag
        For hvert dokument W i M:
            For hvert ord w i dokument W:
                Fjern det nuværende topic k for ordet w

                For hvert topic k fra 1 til K:
                    Beregn sandsynlighed P(k) baseret på:
                        a. Hvor hyppigt topic k optræder i dokument W (Document-topic-fordeling)
                        b. Hvor sandsynligt ordet w er under topic k (Topic-word-fordeling)

                Vælg nyt topic k' til ordet w baseret på sandsynlighedsfordelingen P(k)

                Tildel det nye topic k' til ordet w

        Opdater Document-topic-fordeling og Topic-word-fordeling baseret på nye topic-tildelinger

    Indtil (konvergens nås: ændringerne mellem iterationer er minimale)

Returnér Document-topic-fordeling og Topic-word-fordeling
Slut

Suppose you’ve just moved to a new city. You’re a hipster and an anime fan, so you want to know where the other hipsters and anime geeks tend to hang out. Of course, as a hipster, you know you can’t just ask, so what do you do?

Here’s the scenario: you scope out a bunch of different establishments (documents) across town, making note of the people (words) hanging out in each of them (e.g., Alice hangs out at the mall and at the park, Bob hangs out at the movie theater and the park, and so on). Crucially, you don’t know the typical interest groups (topics) of each establishment, nor do you know the different interests of each person.

So you pick some number K of categories to learn (i.e., you want to learn the K most important kinds of categories people fall into), and start by making a guess as to why you see people where you do. For example, you initially guess that Alice is at the mall because people with interests in X like to hang out there; when you see her at the park, you guess it’s because her friends with interests in Y like to hang out there; when you see Bob at the movie theater, you randomly guess it’s because the Z people in this city really like to watch movies; and so on.

Of course, your random guesses are very likely to be incorrect (they’re random guesses, after all!), so you want to improve on them. One way of doing so is to:

• Pick a place and a person (e.g., Alice at the mall).

• Why is Alice likely to be at the mall? Probably because other people at the mall with the same interests sent her a message telling her to come.

• In other words, the more people with interests in X there are at the mall and the stronger Alice is associated with interest X (at all the other places she goes to), the more likely it is that Alice is at the mall because of interest X.

• So make a new guess as to why Alice is at the mall, choosing an interest with some probability according to how likely you think it is.

Go through each place and person over and over again. Your guesses keep getting better and better (after all, if you notice that lots of geeks hang out at the bookstore, and you suspect that Alice is pretty geeky herself, then it’s a good bet that Alice is at the bookstore because her geek friends told her to go there; and now that you have a better idea of why Alice is probably at the bookstore, you can use this knowledge in turn to improve your guesses as to why everyone else is where they are), and eventually you can stop updating. Then take a snapshot (or multiple snapshots) of your guesses, and use it to get all the information you want:

• For each category, you can count the people assigned to that category to figure out what people have this particular interest. By looking at the people themselves, you can interpret the category as well (e.g., if category X contains lots of tall people wearing jerseys and carrying around basketballs, you might interpret X as the “basketball players” group).

• For each place P and interest category C, you can compute the proportions of people at P because of C (under the current set of assignments), and these give you a representation of P. For example, you might learn that the people who hang out at Barnes & Noble consist of 10% hipsters, 50% anime fans, 10% jocks, and 30% college students.

OPRINDELIG TEKST

• \(P(w|z, \phi)\) er sandsynligheden for at observere ordene \(w\), hvis vi kender topic-tildelingerne \(z\) og topic-word fordelingerne \(\phi\).

  • Når vi ved, hvilke topics ord kommer fra, og hvordan topics “genererer” ord, kan vi beregne sandsynligheden for ordene i dokumenterne.

• \(P(z|\theta)\) er sandsynligheden for at hvert ord får en bestemt topic-tildeling \(z\), givet dokumenternes emnefordeling \(\theta\).

  • Hvis vi ved, hvor meget hvert topic fylder i dokumentet, kan vi beregne sandsynligheden for, at ordet tilhører et bestemt topic.

• \(P(\theta|\alpha)\) er sandsynligheden for document-topic fordelingen \(\theta\), givet Dirichlet-fordelingen med hyperparametre \(\alpha\).

  • Vi starter med en antagelse om, hvordan topics typisk fordeler sig i dokumenter (bestemt af ). Denne antagelse justeres iterativt, når vi ser på data.

• \(P(\phi|\beta)\) er sandsynligheden for topic-word fordelingen \(\phi\), givet Dirichlet-fordelingen med hyperparametre \(\beta\).

  • Vi starter med en antagelse om, hvordan ordene typisk fordeler sig i topics (bestemt af \(\beta\)). Denne antagelse justeres iterativt ud fra data.

• \(P(w|\alpha, \beta)\) er en normaliseringsfaktor, der sikrer at sandsynlighederne summerer til 1.

• Den gør dette iterativt, indtil sandsynligheden for at observere vores data (givet de modellerede topis) bliver maksimal—og resultatet bliver en meningsfuld inddeling af store tekster i skjulte topics.

• Grundlæggende vil vi finde de værdier for \(\theta\), \(\phi\) og \(z\), der gør, at sandsynligheden for de pågældende topic-inddelinger bliver størst muligt (maksimering af posterior-sandsynligheden).

• Document-topic fordeling, \(\theta\): Én tekst består af \(70\%\) “politik”-topic, \(20\%\) “økonomi”-topic og \(10\%\) “teknologi”-topic.

• Topic-word fordeling, \(\phi\): “Politik”-topic er kendetegnet ved ord som “regering”, “valg”, “minister”. “Økonomi”-topic ved ord som “inflation”, “arbejdsløshed”, “vækst”.

I LDA er vores prior, startantagelse om fordelingerne, en Dirichlet-fordeling.

\[ f(\theta; \alpha) = \frac{\Gamma\left(\sum_{i=1}^{K}\alpha_i\right)}{\prod_{i=1}^{K}\Gamma(\alpha_i)} \prod_{i=1}^{K}\theta_i^{\alpha_i - 1} \]

Parametrene \(\alpha_{i}\) er vores “prior count” for topics (som vi selv vælger) og bestemmer graden af variation i fordeling af topics: